
United States FXtent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Addresi: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



| APPLICATION NO. FILING DATE FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/577,478 05/25/2000 Thomas S. Heath 


3351-048 


6587 


7590 08/15/2003 






Lowe Hauptman Gopstein Gilman & Berner LLP 


EXAMINER 


c/o Kenneth M Berner 
Suite 310 

1700 Diagonal Road 
Alexandria, V A 22314 


NATNAEL, PAULOS M 


ART UNIT 


PAPER NUMBER 



2614 

DATE MAILED: 08/15/2003 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 07-01) 



Office Action Summary 



Application No. 

09/577,478 



Examiner 

Paulos M. Natnael 



Applicant(s) 

HEATH, THOMAS S. 



Art Unit 

2614 



« The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )|3 Responsive to communication(s) filed on 22 May 2003 . 
2a)[3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) I3 Claim(s) 7-23 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) [3 Claim(s) 1-23 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

1 1) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)0 All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 1 9(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 . 

Attachment(s) 

1 ) □ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTCM1 3) Paper No(s). . 

2) □ Notice of Drafts person's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) [H Other: 

U.S. Patent and Trademark Office " ' " ~" 

PTO-326 (Rev. 04-01) Office Action Summary Part of Paper No. 9 
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DETAILED ACTION 



Claim Rejections - 35 USC § 102 



1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351 (a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

2. Claims 1,2,4,5,7,11,12,20-23 are rejected under 35 U.S.C. 102(e) as being 
anticipated by De Bonet et al., U.S. Pat. No. 6,510,177. 



Considering claim 1 , De Bonet et al. disclose all claimed subject matter: 

a) the claimed method of extracting a sequence of video frames is met by step of 

Extract l-Frames and motion Vectors From base Layer 910, FIG. 9; 



b) the claimed method of sampling each of the video frames is met by step 915, 
upsample Extracted l-Frames, Fig. 9; 



c) the claimed method of interpolating the up sampled video frames is met by step 920, 
Generate predicted l-frames, FIG.1; 
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d) the claimed method of aligning the interpolated video frames is inherent, because the 
alignment is a necessary step or process, without which the image wouldn't be 
displayed properly. 

e) the claimed method of creating a single image from the aligned video frames is met 
by step 960, Output Enhancement Layer 960, Fig.9; 

Considering claim 2, the claimed method wherein the sequence of video frames are low 
resolution images is met by the disclosure that the "base layer contains low-resolution 
video information" (see Abstract). 

Considering claim 4, the claimed method wherein said align step comprises aligning the 
video images in an x direction and a y direction in a center portion of interest in each 
video frame is inherent, because alignment of images implies in all directions. 

Considering claim 5, the claimed comprising extracting the sequence of video frames at 
30 frames/sec is inherent, because the rate of 30 frames/sec is standard in 
video/television systems, for example. 

Considering claim 7, the claimed method of correlating the up sampled video images; 
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Regarding claim 7, see rejection of claim 1(c). 

Considering claim 11, identifying commonality from one individual frame to the next and 
overlapping the individual frames and displaying an image representing a continuous 
area is inherent because individual frames must be overlapped to be displayed as a 
continuous image. 

Considering claim 12, extracting the sequence of video frames at 30 frames/sec is 
inherent, because the rate of 30 frames/sec is standard in television systems. 

Considering claim 20, see rejection of claim 1 . 

Considering claim 21, De Bonet et al. disclose the following claimed subject matter, 
note; 

a) the claimed at least one sequence of machine executable instructions is met by the 
Application Programs 134, FIG. 1 . (see col. 5, line 49 to col. 7, line 20) 

b) the claimed a medium bearing the executable instructions in machine form, wherein 
execution of the instructions by one or more processor causes the one or more 
processor to extract a sequence of video frames, up sample each of the video frames; 
interpolate the up sampled video frames; align the interpolated video frames; and create 
a single image from the aligned video frames; 
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Regarding b), see rejection of claim 1 

Considering claim 22, De Bonet et al. disclose the following claimed subject matter, 
note; 

a) the claimed processor is met by Processing Unit 21 , FIG. 1 ; 

b) the claimed memory coupled to said processor is met by system memory 104, FIG. 

1 = 

c) the claimed memory having stored therein sequences of instructions when executed 
by said processor, causes said processor to perform the steps is met by the 
Applications Programs, 134 FIG. 1. 

d) the claimed steps of extract a sequence of video frames; up sample each of the video 
frames; interpolate the up sampled video frames; align the interpolated video frames; 
and create a single image from the aligned video frames; 

Regarding d), see rejection of claim 1 . 

Considering claim 23, see rejection of claim 1 . 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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4. Claims 3,6,10, 13,14, are rejected under 35 U.S.C. 103(a) as being unpatentable 
over De Bonet et al. U.S. Pat. No. 6,510,177. 

Considering claim 3, the claimed method wherein said up sample step is by a factor of 
4; 

Regarding claim 3, De Bonet et al do not disclose whether the up sampling is by 
a factor of 4. However, it would have been obvious matter of design choice to modify 
the De Bonet reference by having an up sampling of factor of 4, or any other factor, 
since applicant has not disclosed that having the sampling factor of 4 solves any stated 
problem or is for any particular purpose, and it appears that any up sampling factor 
would perform equally well with the disclosed sampling method of De Bonet. 

Considering claim 6, the claimed wherein the sequence of video frames includes 5 
video frames; 

Regarding claim 6, De Bonet et al do not disclose the actual number of frames in 
the sequence. However, it would have been an obvious matter of design choice to 
modify the De Bonet reference by choosing any number of sequence of video frames, 
since applicant has not disclosed that having five video frames in a sequence of video 
frames solves any stated problem and it appears that any number of frames would 
perform equally well with the disclosed up sampling method of the reference of De 
Bonet. 
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Considering claim 10, the claimed aligning each the extracted sequence of video frames 
before said up sample step; 

Regarding claim 10, see rejection of claim 1(c ). 

Considering claim 13, the claimed wherein the sequence of video frames includes 5 
video frames. 

Regarding claim 13, see rejection of claim 6. 

Considering claim 14, the claimed correlating the up sampled video images is inherent, 
because it's a necessary step or process during sampling and/or interpolating of the 
video images. 



5. Claims 8,9,15-19 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over De Bonet et al. U.S. Pat. No. 6,510,177 in view of Szeliski et al., U.S. Pat. No. 
6,018,349. 

Considering claim 8, the claimed averaging a pixel intensity from each of the up 
sampled video frames; 

Regarding claim 8, De Bonet et al. do not specifically disclose averaging the pixel 
intensity. However, averaging pixel intensity for data picked up by cameras would be 
obvious to perform in video processing so that the image quality is enhanced further. 
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In that regard, Szeliski et al. discloses a patch-based alignment method and apparatus 
for construction of image mosaics, wherein Szeliski specifically teaches that The 
blending of the corresponding pixel values employs a weighted average... Each weight 
Wk is proportional to the proximity of the image pixel location Xk to the center of the 
particular image Ik..." (col. 29, line 58 through col. 30, lines 6) Therefore, it would have 
been obvious to the skilled in the art at the time the invention was made to provide pixel 
averaging and modify the system of De Bonet in order to further enhance the quality of 
the video data. 

Considering claim 9, compensating for platform movement and rotation zoom. 

Regarding claim 9, De Bonet et al. do not specifically disclose compensating for 
platform movement and rotation zoom. However, such compensating would have to be 
made in order to prevent images from being distorted and other artifacts introduced in 
the composite image. 

Szeliski discloses a patch-based alignment method and apparatus for 
construction of image mosaics. Szeliski teaches that "Based on this rotational 
panoramic representation, block adjustment (global alignment) and deghosting (local 
alignment) methods disclosed herein significantly improve the quality of image mosaics, 
thereby enabling the construction of mosaics from images taken by hand-held 
cameras." (Col. 32, lines 13-9) Therefore, it would have been obvious to the skilled in 
the art at the time the invention was made to provide for platform movement and 
rotation zoom compensation and modify the system of De Bonet in order to prevent 
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images from being distorted and artifacts introduced in the composite image and 
enhance the quality of the video image. 

Considering claim 15, see rejection of claim 8; 
Considering claim 1 6, see rejection of claim 9; 

Considering claim 17, the claimed aligning each the extracted sequence of video frames 
before said up sample step is inherent, because alignment is a necessary step or 
process, without which the image wouldn't be displayed properly. 

Considering claim 1 8, see rejection of claim 11. 
Considering claim 19, see rejection of claim 12. 



Response to Arguments 

6. Applicant's arguments filed May 22, 2003 have been fully considered but they are 
not persuasive. Response follows: 



Applicant's Arguments 

a) The Examiner has failed to identify all elements of claim 1 as anticipated by the De 
Bonet reference. The Examiner asserts that the De Bonet reference describes 
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interpolating the up sampled video frames: however, there is no description of 
interpolation of up sampled video frames to be found in the reference. Specifically, the 
De Bonet reference solely describes step 920 in terms of the result of step 91 5, the up 
sampling of the video frames. The interpolating step of claim 1 is not to be found in the 
De Bonet reference. 

b) That is, because the De Bonet reference does not create a single image, rather 
multiple video frames are produced, there is no need for aligning the video frames. The 
present invention requires alignment of the interpolated video frames because a single 
image is to be created based on the video frames. For least this reason, the rejection of 
claim I should be withdrawn... "The enhancement layer includes a high resolution video 
stream." (De Bonet at column 7. lines 32-33). ' The enhancement layer output by step 
960 is a high resolution video stream and not a single image as claimed in claim 1 

c) With specific reference to claim 1 , the Examiner asserts that the claim limitations are 
inherent because individual frames must be overlapped to be displayed as a continuous 
image; however, the Examiner has failed to identify where in the reference identifying 
commonality from one individual frame to the next is performed in the reference. 

d) The Examiner's attention is directed to the present specification, specifically to page 

1 1 , lines I-5, wherein the specification describes the factor of 4 as found by the inventor 
to be optimal and, yielding the most consistent results. Further, specific benefits of using 
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a factor of four are described, as well as reasons for not using a higher factor. For at 
least this reason, it would not have been an obvious matter of design choice to modify 
the De Bonet reference to use an up sampling factor of four as asserted by the 
Examiner. 

e) The Examiner has failed to identify any motivation or suggestion in either reference 
teaching, suggesting, or describing the asserted combination. The Examiner appears to 
have improperly applied hindsight reasoning based on the present invention to make 
the asserted combination. 

Further, the references operate in a different manner and produce a different 
output. That is, the De Bonet reference is directed to video frames of a video stream, 
while the Szeliski reference is directed to a combination of still images to construct a 
mosaic. 

Examiner's Response 

a) De Bonet et al disclose a system and method for layered video coding enhancement. 
De Bonet et al. disclose a base layer and an enhancement layer. The latter contains 
high-resolution information, and provides a variable amount of enhancement to the base 
layer. The enhancement layer module shown on FIG.9 discloses the process of 
extracting l-Frames, upsampling the extracted l-frames, generating predicted l-Frames 
(step 920), calculating a residual frame for each prediction (step 945), as well as 
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performing pre-compression processing on residual frames (step 950). Steps 920, 945 
and 950 are types of interpolation. 

Interpolate = to estimate values of (a function) between two known values. 
(Merriam Webster's Collegiate Dictionary 10 th edition) 

Specifically, De Bonet teaches that "pre-compression is an optional step that 
modifies residual values in the residual frames so that some parts of the residual are 
amplified and others are attenuated. The modification is based on an assessment of the 
visual importance of each residual value." (col. 14, lines 7-10) Therefore, the argument 
that "there is not description of interpolation of up sampled video frames to be found in 
the reference" is considered unpersuasive. 

b) De Bonet discloses that the enhancement layer 228 provides a high resolution 
information. And then, the "encoded, layered video signal (not shown) is sent to the 
transmitted 230 which sends the signal vial a satellite 240.... The decoded video signal 
is then sent to the HDTV set 290 for viewing." (col. 8, lines 37-56) This disclosure 
suggests that the video signal is output as one video signal from the encoder to the 
satellite transmitter, and then received by the receiver to be displayed as one signal on 
a display such as the HDTV television set. Therefore, Applicant's argument that the 
signal is a high resolution video stream and not a single image as in claim 1 is not 
persuasive. 
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c) De Bonet discloses that "When applying motion vectors to the high-resolution I- 
frames and P-frames the boundary problem can be reduced by using overlapping 
motion blocks... In effect, using overlapping motion block prediction "smoothes" the 
boundary of adjacent macroblocks." (col. 15, lines 19-31) Identifying commonality from 
one individual frame to the next and overlapping the individual frames and displaying an 
image representing a continuous area thus is inherent because individual frames must 
be overlapped to be displayed as a continuous image, and inherent because if the 
frames are not the desired frames, i.e., if the commonality is not identified as belonging 
to the same process, the process would be flawed. Argument therefore is not 
persuasive. 

d) De Bonet et al disclose do not disclose whether the up sampling is by a factor of 4. 
However, it would have been obvious matter of design choice to modify the De Bonet 
reference by having an up sampling of factor of 4, or any other factor. As Applicant 
admitted, a sampling factor below 4 may also be acceptable. Thus, any sampling factor 
desired by the designer would perform equally well with the disclosed sampling method 
of De Bonet. 

e) Although, averaging pixel intensity is not disclosed by De Bonet, such processing for 
data picked up by cameras would be obvious in video processing because of the need 
for image quality enhancement in video processing. 
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Szeliski et al. discloses a patch-based alignment method and apparatus for 
construction of image mosaics. Specifically, Szeliski teaches that "The blending of the 
corresponding pixel values employs a weighted average... Each weight Wk is 
proportional to the proximity of the image pixel location Xk to the center of the particular 
image Ik..." (col. 29, line 58 through col. 30, lines 6) 

Thus, the skilled in the art would readily recognize the need for such processing 
technique to enhance image, and modify the system of De Bonet to enhance the quality 
of the video data. Hence, the argument that the assertion by Examiner was a hindsight 
reasoning base on the present invention, is not based on fact, and it is unpersuasive. 

Similarly, although De Bonet et al. do not specifically disclose compensating for 
platform movement and rotation zoom, however, such compensating would have to be 
made in order to prevent images from being distorted and other artifacts introduced in 
the composite image. 

In that regard, the Office Action proposed the combination with Szeliski which 
discloses a patch-based alignment method and apparatus for construction of image 
mosaics. Specifically, Szeliski teaches "Based on this rotational panoramic 
representation, block adjustment (global alignment) and deghosting (local alignment) 
methods disclosed herein significantly improve the quality of image mosaics, thereby 
enabling the construction of mosaics from images taken by hand-held cameras." (Col. 
32, lines 13-9) 

Therefore, it would have been obvious to provide for platform movement and 
rotation zoom compensation and modify the system of De Bonet in order to prevent 
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images from being distorted and some artifacts introduced in the composite image and 
ultimately enhance the quality of the video image. 



Conclusion 

7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Paulos M. Natnael whose telephone number is (703) 
305-0019. The examiner can normally be reached on 6:30am -3pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John Miller can be reached on (703) 305-4795. The fax phone numbers for 
the organization where this application or proceeding is assigned are (703) 872-9314 for 
regular communications and (703) 872-9314 for After Final communications. 
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Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 



4750. 




PRIMARY EXAMINER 

Paulos Natnael 
August 1 , 2003 



